esp@cenet document view 1/1 K — > 



METHOD AND DEVICE FOR SUMMARIZING VIDEO IMAGE 



Publication number: JP2001 11 9649 
Publication date: 200 1 -04-27 

inventor: ECHIGO TOMIO; TOMITA ALBERT; MASUMITSU 

TAKESHI 

Applicant: IBM 
Classification: 

- international: G06F17/30; H04N5/76; H04N5/91; G06F17/30; 

H04N5/76; H04N5/91; (IPC1-7): H04N5/76; G06F17/30; 
H04N5/91 

- european: 

Application number: JP1 9990300349 19991022 
Priority number(s): JP1 9990300349 19991022 



Report a data error here 



Abstract of JP2001 1 1 9649 
PROBLEM TO BE SOLVED: To generate a 
summary of video images along user's 
preference. SOLUTION: This device generates 
a summary from video images and is provided 
with a means that extracts numerical data from 
objective video images, a means that decides 
an important part of each scene on the basis 
of the extracted numerical data and a means 
that generates a summary from parts of the 
importance, that have a value which is a 
prescribed threshold or higher. 
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(57) [fi»] 

[SBK] *irail§:MKfcL«fc5iTSWBtt. a— »f© 




<t ft swfc^^fttt^- * ^jmrn-r s x r- ? ?'i . 

tCl-J^tg^-XDlSg^^t,, BUlBS^ift-S 

MM i ft *W**p&**:F- f ^a&tu-r & X t- v . 
i. 

*fcR i ft S8MR*>6Sfi'? r - $ 5riJUT5X r s» ?'i . 
fiiajfflffl hidgm?- £fcs~3X>r&->- >©figR£ 
atttS^f 1 ? ?"i. 

HulBfPfiS § nfcg^^r^-r 4Xf >7i, 
ITSXf v7i, 
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itlitB^ i ft 4z v >r • x-££^t?*§ 

teS-3t,>T&'>->©mgg/&&5£U B5fB>tfftift-5> 
^^•fev>-r-< • r-££^£ftt>*§^icfi, 
10 IB3fcffif r -£&Cgr5t,>T&'>-->©fiSg£:&S-rS# 
Si. 

>©SSS=£^-r i . 

*r^i ft sB^^eafiif 1 - z zmm-r , 

i ft S Wfc£*jf:it>CC 3.- If <D0 z * 

BufBWtH UtcWtmf- *RS>*BulB^--tf©*f^4^-r -r 
- £ {Cg-3t,*-C§^- >©SSS4^-T i . 

«fiasss3&ss?s©a<i«±©sp^*> eg^&ms-r & 

30 ^gi, 

i ft 6 Ifeftfr 6 ttfflf 1 - * *nna - j- 4 <t . 
iJiamss tifcSR>*^-r5^si , 

iriBSi^^^tftc i-if ©A^JCCfc 0 t5IBfiSS4^ 
40 JEfS^Si. 

"C, 

fjiBi^x-^Ati, 

A*$n/cBft#.4IBtf.T^lBl«^gi . 
K#.S^«Si*^, 

so ^taift-sift^e^tfii^-^^jaai-r^^i. 
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ifimomiixmil 

[0 00 1 3 
[0 00 2 ] 

[fi£*©««] 2ooo^©BS^#ju-^utr»as 

Kit**, i±^f^^^S^2 0 0 3$K:J^^^ 

[0 00 3 ]«©7tP^Mt(i, JX-lfraUTJl/ 
H/ctfft • 7 S ^*(CR«*»BIL. «ffllBB*S«Flffl<D 

[0004] *»i8CD»«i&swH*ttt, B#ra^i4> 

[0 00 5 ] Urt>U mZfrtkMtt&RTZCZtc 

ffi«£T*SS<D^- K * X ( H D D ) # 

5«»T*£-fe ( S e t - T o p 

Box. STB) ^cffl^iA^n, :i~lf(DJSF#K:J:-5> 

[0 00 6 ] CCX\ hbv-f'tfv** (ST 
B) ^r-^TV (CATV) <DiMKBSfE-t>. il 

[0 0 0 7 3 *#-:?8fe»*ifc**»6ja**i 

< ft t*0»«R*3i*s«A fctiTl* 5. 
[0 00 8] fi^OIMtiJI&Sm a 

[0009] cctse^u, *rfti3te*Kfa^6«Fpa 
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!^^6^-1f©Pltt©ttt^-^>^*B»l!ftL. Pitt 

[0010] 8Mfc©*fo«:«:. rtSUWftjfiWirSte 
to, *tttttt^-y^JU- :n>fc:^-£ (PC) 

to, Efi^ffiSTSfflRO^-^Sr^-^ttairai 
0, :z- If # @ fiCC ^r - £ ££l#^bi*T ^ -If {I * CC 

[OOllJJfc OMNBIMKtlrrSSTBTlUI 

[ooi2] «e*^6o»«©Bjiftaa5i it, 
■r->-> • 5 L ^>^<z)5feHiia^ ; &a£^/cx h-y - • # 

— K (story board) «r*tf rc^5ffil^*5. 
[0013] Xr-y- • ^FgWi^-> 

[ 0 0 1 4 3 X h- «J - • F©H«<Da(HaE«:«: 
20 to, M(l^77r-a(D^->?:i8L/ci/-> • h7 

>> p > / 3> • ^* :7 7 (Scene Transition Graph) [1] 
Tm. Yeung, B.L. Yeo, etc ., "Extracting story units 
from long proqrams for video browsinq and naviqat 

ion, "IEEE ICMCS, pp. 296-305, 1996. J lit. tyk{£<DW 

[0015] mmmffixtz, tr^* -^^>^ (vi 

deo Skimming) [2] fM. Smith and T. Kanade, "Vid 
eo skimminq and characterization through the combin 
ation of image and lanquaqe understanding technigu 
30 es," IEEE CVPR, pp. 775-781, 1997. J iW&WtfcV}Xt£ 

[0 0 1 6 3 *gtC*$l>-Cte, STB^fSLte&ris 
Y) , w y • f^X^I (HDD) *WTSS«Flfflia 

HfnrfiEtt««*s«asnr*jo. Mb tv , Tivot 

TJl-rfy XA£LT. ^^(Bays)S^^t 
t 4 5 ( [ 3 ] r^TV, Preference Detenrri nation Enqin 
40 ej ) 0 

[0017] Cft^WSMfc* A hbffirll 

[o o i 8 ] ^zmzm^zt. MM^cjz^rm^ti 

[0019] KfROrtgi^TS^-^CD^^r/tL 
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$ (semantic data) £Vi;1gtlCi$l1Z>{m<D%i$ 
SMi'r— £ (numerical data) #&£ 0 
[0 0 2 0] -i2^>-r * * ^ • ^-^KBMlMc^StiS 

tO 02 1] CCCiz-^itt, -«WCC«i»fB3|R*^ 
[0 0 2 2] ^-If^cboT y ^ • f*— £ 10 

[0 0 2 3] -fev><7W ? >7 . -f-^ttlRft^-^^jB 

[0 02 4] Ig-fe7>r * v >? • A*fc J: 

[0 02 5] £TG>R«MC, ttR©*** -fev>fw 
[0 0 2 6 ] i'Oct^^if-^^T^^^t 

y*A(DRfl«BS%H3erSC<!: 30 
[0 02 8] *#BM«C <t 0 , fi«FlHO!!fc«*»€).3.-1f*s 

[o o 2 9 j Rfcecttw-rs^-frtt, ^-^ttas-css 
[0030] w^tcftw-rs^-^s&s+^-cttc**^ 

[0 0 3 1] 
[0 03 2 ] 
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[0 0 3 3] 

[0 0 3 4] attrtt, ttaS»tt!SMfcW:«H[-r442v> 
[0 0 3 5] a— 5f#^aE<0IB«C, **M*«ftUfcS 

tb», (An) -i-sictfccaiirfe 

o-c5E»b3fcBMMaa*jiff l (out. mttm, t>s 

[0036] gtcmmtc. imfrhftmisx&zmAffi 

[0 0 3 7] ^OttW6»l»©tttiBft3&$3infc»^ 
[0 0 3 8] a— WM^^iSTBK, ^00 

>w^<t 2:^7^5 • ?^7t,ci£mLxm& 
[0 03 9 ] ttiMW^KiRCcfrfiacrS-fev^f'^ 

[0040] *to. ^-^mA<ommm^<k^6^ 
*j*£Sfrr-**v xuRtRopso *fflufcfflA<z>»fi 

[0 04 1] tf>^;br«, «Utt«B«(D-35cy 

^^r^^e>n, -e-ofit^o— i ^T'O^Hii u Ki* 

*SSES:Sf^B# Palate > £rfS£-r o a«*s#W 6 

nri^ (Hi). 

[0 04 2] JJtfCJL-lftt, ^ ^ ^ . -r-^^ 

[0 04 3] i-1f*S|MR**»t>* i, ftffOWKI 
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(X i ) <Dffi£k i tUcit, Xi^ki 

[0 04 5] ^(CffeCD^EfflraSffi^-^ (X i ) CDfB 

(X i ) CCjtfT£;^^-£ ■ ^f>>'t;Wi@2(DJ: 

[0 0 4 6] y-f^*^HW»K:*JWSJn«a»SiX 
i CcteWS'^y — * • 3^f*>t^ + ;i/©JlS«l3&so <bte& 10 
BBfflTh4*to, M'^y-* • #<r>*>*Jl/*0^ 1 liC 

[0 04 7] t&CD&mr-Z (Xn) fcoHTfeHtt 

[0 04 8] jSPJWaKS^/hSirii*, rc^-rf-*-# 

miitL. *Ot*il«Thtt/hS<«t4. 
[0049]8Htf5#, KR*««)5it#f>'> 20 
*^fiiWllCDll©»fO*««&4ft:«>. SfettfbLtaf*?- 

[0 0 5 0] *fc. ttfl^-dr^WRmtr^&it^tt. tt 

[0 0 5 1 ] B— 7U-ACC«5»Sft{Bf r -^ (X i ) 
i (Xj) <Dtt*s*ti-e*i k i , p j <D£ * % ^piftW k 

i. p j r»lfcWaA2 ©2*5BEa«**ap»'r*. 

[0 0 5 2] ^^>^ + ;KDJ6«]3&S0CC&SJ:5tcBlffl 
Th2£3£a6 k 2&7C#^>5/ + Jl>- #57*0^ 1 &C 30 
IE*I<i:ir& 0 ni»©»ttf-*{C»Otn (n-1) 
/2 02^7n^f->^ + ;l/ • ^7*J*J5Kr*4. 

[0 0 5 3 3 —*. ^-1f*stt«*X*9^Ufct#, 

[0 0 5 4] CCtMfii^ — ««CJH»V^^A 

[0 0 5 5] X* * y'£tltc&mc£gtl2> m&f- 40 
MXi) CDfil^k i tUtctB, X i # k i £jp 
#K a£r#tfc<bU k i <Dtf^>WJUp < k i ) tcfcf 
Op (k i ) -/2flS©iEJS»fl3%S-rSo 

[0 0 5 6] ffeOSMI^-- * (X n ) Rtf. a»(D$*ffi 

[0 05 7] • ^f*>^ + J^O'2^ 

[0 0 5 8] S^U-Atcfc^SKftOKffl-r-^© 
(^f>ytJbffi-Th) R^2^7C*-f*>^^Jl/ • 4/ 50 



#H2 00 1 - 1 1 9 649 
8 

77©(!i<f>'>t;HI-Th2)©IBn?:*a), S/^ 
[0 05 9] *s9*4 FW»«::fott£T = c/S (c : 

[0 06 0] «±©J;^(CA7>-^ ■ #^>^*Jl/£ 

[0 06 1 ] J/c, ^-1f», ^f»til/-^7 7^ 
[0 06 2] -te v>x o^' — * te—grT £ 

[0 0 6 3 ] COi*©A*tt«*>#^>^*^ • ^7 

[0064] vL±&&mM<Dmm<Dmti<DWt^x*foi) t 
eiTrH*ffli»TH«:B«B«:iBi?Hr*. 

[0 06 5] B3tt, *»?8<Z>^ft©iStttl«:7nU/c7 D 
[0 06 6 ] Af^7'l 1 Otcfcl^r, ^ggOjffiStf 

[0 06 7 ] x^v yi 2 occ&t^r, msRomaw 

[0 06 8 ] ^ 1 3 Otc^Jl^r, SfficD^SO' 

a - if ©aiFygm^tf *>n & . 

[0 0 6 9 ] Xf ^7*1 4 O&CfcHT, ^-rf-$r • # 

»^«, ^7*1 7 0 CC^f L . . 7f<r> 

S^ + ;KOM»fttftottC^ai^tt, Xf-;7'1 bo^mf 

[0070] Af77'17 OSCte^T, «tffc¥Wffi*ffl 

[0 0 7 1 ] C<D^7^-£ • ^>^-WU£JESrE£, 

[0 0 7 2 ] Af^l8 oic*$i>-c\ — $ - # 

[0 0 7 3 ] 3.— !f©^7^-^ • #^>Sx*;basffifiE 

[0 07 4] Xf 7 ^ 1 5 OCCtet^T. U-A^ 

[0 07 5] 09x.«. lo<D3>f>74t^bo 
/cit»7iT5C<!:mS„ 



t 



(6) 



[0 0 7 6] Xf"> 7' 1 60m^, &y U-A£EX * 

oau s^fv i 2 o^tT*rs B 

[0 0 7 7] Xf-; 1 9 OdOs^X. S^S^TT 
S. 

[0 0 7 8] H4B. B3©KBa»tti#Hl Ur'^' 

[0 0 7 9] Xr^ 7*2 1 OtC*Jl>T* SBS#WA5ffi 

[0 0 8 0 ] X^»>y*2 2 OCC*Jl»r, ^ U-ACC*J 
^SSIf-^X i (i = l, 2, • • • n ) CC:fct>T*10 

sumjy - £ ( p(X/) - T/*_/ ) + 
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^ti^np (x i ) -Th_icDffl^ftmr^ e 

[0 0 8 1 ] Xf^ ^2 3 OtC*$l>T. 31:7 U-ACCte 
WftKH^-jr Offl (X i , X j ) (K=i, j< 

(Xi, Xj) -Th2_i j (DfflteiWT*. 
[0082] Xf77'24 OKl&UT. ^T77"220 
SO' 2 3 0Vn^ntc±X<Dm<Dm ( s um_p ) ^ 

[0 0 8 3 ] 
[Kl ] 

E ( p&txn - T«jf > 



[0 08 4] ^7*2 5 OtCtel^r, hi 
[0 0 8 5] CCt, *?9*A K««f*JKTO)3a5-C#it 
[0 0 8 6 ] 



/>(X) = 1- 



cxp(sum _p/T) 



[0 0 8 7 ]i5lt O 3 Otlift^R^^-lf CDjltR 
* HSBffiffl (Xt^> 7'130) £M&Cf$«ci&Bj§o/c:7 

[0 0 8 8 ] Xt^:/3 1 OtCfc^T, HftSTSSV^ 

-if" ©as? • ®mmm Uf^i30) tfHf&sti 

[0 08 9 ] Xf77*32 0^t, W&*ff5*»5 

2k Sfa*tT*D&C>»^ra:. x-r ^3 4 0 tcittf. 
[0 0 9 0 ] Xf ^330tcfe^t, ISS^HffS© 
HfgTh.I mpJiOi^tt, CO^-xaiTft 

[0 0 9 1] -J, MSS^gf^OUfflT h_I m p« 

4 0 ^jttf o 

[0 0 9 2 ] Xr>;7'34 0(C*5liT. U-ACDH 

[ 0 0 9 3 ] Tsv-y 7*35 O^Cfcl^r, 3StIiffi£J|-Cl> 
*:x— !f#, CO^>->€:Bfcli (Accept)^ 
jSfc<&C» (Re j e c t ) i)\ ftt,>tt, df^^ferfe 
^(neutral) frOBiR *W 9, 

[0 09 4] C(D*iJ»r0Je*, J&fcl» (Accept) 
Xtemc< ^(Reject) <Dt§^£, Xf^7'3 
6 OlcMfr, £%6fc-Cfcttl» (neutral)©! 
X-r ^3 7 O^jit*. 



[0 0 9 5 ] X-f^ 7*3 6 OOCfclvr, ACD£fc 
m-r-ZX i ( i = 1, 2, • • • n) *Jifcl> (Ac 
c e p t ) XttJ|fc< ftlr* (Reject) QMM^ft 

[009 6]@6it H3©3fi{fc^gCcJ:S^.y-# 
20 • sPf-^^ + iHeBrMa (^f^l 7 0) fcjgtCSMffl 

[0 09 7 ]Xr^7'41 Otefcivc, ?fi{b^S0c<i:S 

[0 0 9 8 ] Xf77*42 0fC*a>T\ &Hr-~£ (tt 
fil-r- >Xi (i = l, 2, • • • n ) ) £EX 9 ffi 
To 

[0099] xf7 7*4 3 otcbivt, moffi^n/cK: 

JK^-^tfJifc^ (Accept) X«Jifc<ttl> (R 
eject) ^spJBfStl, H,fal> (Accept) 
30 £tt % Xf^4 4 0^M t H/t< (Re j e c 
t ) Xf* * "7*4 5 0 ^Jlto. 

[0100] 77*44 OlCte^X. X i 6Dffi#k i 
t?Z>t, ¥^ki, ^»a^ 2iTSiE«^fljOffiSr 
X i (D'^jt-Z • tff^^-WUdJQftU i = 1-n 

[0101] RUSK, (Xi, X j ) ©ffij^ft-eft 
(k i , p j ) ¥*(ki, pj), #ffca 

" 2<D2^X7ZiEm^^ (X i , X j ) CD^T->^i-;b 
• ^:7K:»imu ^t(Z) ( i , j ) ( i = 1-n, j 

40 = i-m) nicest lx, mbftmzmviE.?. 

[0 102] Xf 77'45 OtCfct^T, X i Offl^k i 
<tT5£, Wki, ^ail, k i <£>#ir> 
p (ki) CC»UTp (k i ) /2<DEMftft*X i cd 
— £ * ^-r>5>i>;l/^6^D, i = 1 ~n £T[h] 

[0103] eiB^c, (x i . x j ) 0#^^n-en 

(ki. P j) iTSi, ?ft (ki., pj), »a 
£U ( k i , p j ) CD^-r>^^;i/p (ki, p j ) 
tCftUX, p (ki, p j ) /2<D2'X7tlEmfrJii%: 
50 (Xi, Xj) <D#f*>i/+)l> • ^77^eiC, ±r 



(7) 

11 

<o ( i , j ) corniest lt, ncxn^aiQiKr. 

[0 104] Xf7 7'4 6 OtCfc^T. ?#*E>n/c:£T<£> 
fC&£J; ^KHfiTh l_i 0 - 1 fciESHb-r 

[0106] Xf-; ^4 7 OtCfcl^T, 3£fb3*BK:J:£ 

[0 1 0 7 J B7£tt k *»Wc|8SCC»Lfc:^- K«5 10 
x.T<D— ^JiUTP VR (persponal Video Recorder) 

[0108] ^P^BlOlt t^BaSCT**), 

[0109]7'D^520^ **M*affll/fcy^ 
[0 110] ^P7^53 01J, • ;< 7 7 7t4 

0, »Bffi^S^i^rsKsn/cfcr7 r * 3 £--«FWtc«?? 20 

[0 1 1 1 ] t'ftf-^5 0 5tt, PVROt'f+f 
fS3$5 1 OCCA^SJa. <S#£*i£ 0 
[0 112] t^:TM«B5 1 (Hi. A*3n/cfft 

[0113] tr^sasps 1 otcEttsn^tr^ • 

[0114] sfiStifcef* • tr^tf - 

7 7 7 5 3 O&c— fi$WtciBttStx^-1f3^#arS<b# 30 

[0115] — t^* WWteBttSftfctf 5 s * • 

[0 1 1 6 ] 1 Otc«, Cti*-CK93Ufc#* 

[0117] *jfe|8©rL— tf - -{>^7x-XiUT 
«, ^8{C^L/cJ:9^^ ^ > F^8 0 0^#^6ti 

[01 181 ^^F^SOOlt ^kWM^mM 8 5 40 

o, ¥mm<DW&B 1 0, ^^<z>fi^8 2 owz^ 

i? e 

[0119] FStartJ (^^-h) ^^>804 
5 0 Jca^SJ&en, "Stop: r 

>8 0 et^y ^^rsci^<fco^7S*©efc«^»ih 

etl. ^Pause. (^-X) ^>8 0 8%^y 7 
[0 120! if. ¥^J5mz. 61T©J: 9tt«a*fr 

50 
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[0121] Bffitjsaasfc. a.— tf«^©iSEi^*^ 
^>8u*^y ? ^*r£ 0 

[01221^/c, TUpdateJ (H*f)8 16# 

[0123] • 5j<^>^*;l/gfr£:3.--»fO: 

&HHg<DS^g8 1 8CD£fcfiI£ 

[0124] ^CC, «T©<t o te«iffl*tf 

[0 125] 7^t«^>^ TOnj 822 

[0126] RSK«8 3 0 ttgft*SS*7SU Sl>SB 

[0127] -rtctofe. Sft4*SfS»^K:tt, jRl* 

[0128] MCC, K*»*4ifea— !ftt r A c c e p 
t J ^>8 1 2X« fRejectJ #£>8 1 4 £ 

[0129] SJRjffl©«J§88 2 0*Ctt % Hfii8 26^S 
#W (%) 8 2 8«r«^r-5C£fclIt6T*4 D 
[0 13 0] Hfii8 2 6«, IfiibSffif^>>->0 

[0131] (%) 3Pt*R»0«S«M£ 

£ 0 

[0 13 2]19tC« 1 EI8CC^Lfc1f *#-(Dfiffl^ 
[0133] H 8<D4 5 ftf ? ^-©K^SStSa? 

[0134] 1l J*?*:-: is a >± L/t{i, ^> (^&CD 

[0135] S9&C#^^&— >a>0D^*> (S^Ofp] 

[0136] m#9 1 2«. ^*>CDfS]^ (ftff) ^ 
^WJ*sftS-9 0St?36«l3&«fta[-9 0ST*S D 
[0137] ^EM9 1 ^0(7)ftg^£Urg 

[0 1 3 8 3 ^7^-$ - ^-r>^-^;U9 1 0tC4b%i 
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t, i*«s«9 2 ob, saifr©^^^-^ • 
[0139] ««»«ttS*iLt:tt % «^:®S«a«cDSi 

a ( S ) % S*ffi»««©fe ( r - g . b - y ) <E>ffi(D 

[0140] HQecMiN5MMiiia)^mmo9a 

(S) iCr)t^t(DA7^-^ • ioU 9 3 OifiTjk 

[ 0 1 4 1 3 8M*9 3 2 B, S^®«^CDM» ( S ) 

[oi42] taw 934 #«^ffi««ja©ffis 
(s) ccstjsixr»ffi%*orii5 0 

[0 1 4 3 ] • ^<^>>'^;U9 3 Otctet,* 

20 

[0144] ^fetffgch LT\ R (*> , G (ft) , 

[0145] H9tC^fettfgG (») CCOl>T<7V^ 
— £ • ;U9 5 0^i$nti^ o 

[0146] ff$A9 5 2B. ¥#Jfef##RG (ft) 4* 

[0147] tfM9 5 4B, G (ft) CC» 

[0148] • jJ<<f>^*;l/9 5 0&ctel> 

1£KI9 6 0B k &Illi«<D^y--£ • ^-r>V 30 

*S9 5 6 B&RgMV^^-* • ^f^tWSLt 

[0 1491 J/c HCC±IEQ£lt8 0©^y~#<D 
*BMtt2 8«BH<Dfflfe*lflasti4. 
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